Evaluation system, evaluation device, evaluation method, evaluation program, and recording medium

ABSTRACT

An evaluation system is a system configured to evaluate coverage of an evaluation target by using a captured image of the evaluation target. The evaluation system includes: an image acquisition unit configured to acquire the captured image; a correction unit configured to generate an evaluation image by correcting the captured image; an evaluation unit configured to evaluate the coverage based on the evaluation image; and an output unit configured to output a result of the evaluation carried out by the evaluation unit, wherein the correction unit extracts an evaluation region from the captured image based on the size of a dent region included in the captured image and generates the evaluation image based on the evaluation region, and the dent region is an image of a dent formed on the evaluation target.

TECHNICAL FIELD

The present disclosure relates to an evaluation system, an evaluationdevice, an evaluation method, an evaluation program, and a recordingmedium.

BACKGROUND ART

In order to improve the strength of machine parts, etc., the surfaces ofmachine parts, etc. are sometimes subjected to shot peening treatment. Acoverage measuring device, which evaluates the finishing degree of suchshot peening treatment, is known. For example, Patent Literature 1discloses a coverage measuring device, which calculates coverage basedon an image obtained by capturing an image of a treated surface anddisplays the coverage.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Unexamined Patent Publication No.2011-152603

SUMMARY OF INVENTION Technical Problem

Shot media having various sizes may be used for shot peening. Therefore,the sizes of dents formed on a treated surface change depending on thesizes of shot media. However, if the surfaces having the same area areused as evaluation targets with respect to the shot media havingdifferent sizes, the sizes of the shot media may affect the evaluationof coverage. For example, if the surface of an evaluation target doesnot have a sufficient area with respect to the size of shot media, theinfluence of a single dent on coverage increases, and evaluation ofoverall (average) coverage with respect to the target may fail.

In the present technical field, it is desired to improve the evaluationaccuracy of coverage.

Solution to Problem

An evaluation system according to one aspect of the present disclosureis a system configured to evaluate coverage of an evaluation target byusing a captured image of the evaluation target. This evaluation systemincludes: an image acquisition unit configured to acquire the capturedimage; a correction unit configured to generate an evaluation image bycorrecting the captured image; an evaluation unit configured to evaluatethe coverage based on the evaluation image; and an output unitconfigured to output a result of the evaluation carried out by theevaluation unit. The correction unit extracts an evaluation region fromthe captured image based on the size of a dent region included in thecaptured image and generates the evaluation image based on theevaluation region. The dent region is an image of a dent formed on theevaluation target.

An evaluation device according to another aspect of the presentdisclosure is a device configured to evaluate coverage of an evaluationtarget by using a captured image of the evaluation target. Thisevaluation device includes: an image acquisition unit configured toacquire the captured image; a correction unit configured to generate anevaluation image by correcting the captured image; an evaluation unitconfigured to evaluate the coverage based on the evaluation image; andan output unit configured to output a result of the evaluation carriedout by the evaluation unit. The correction unit extracts an evaluationregion from the captured image based on the size of a dent regionincluded in the captured image and generates the evaluation image basedon the evaluation region. The dent region is an image of a dent formedon the evaluation target.

An evaluation method according to further another aspect of the presentdisclosure is a method to evaluate coverage of an evaluation target byusing a captured image of the evaluation target. This evaluation methodincludes: a step of acquiring the captured image; a step of generatingan evaluation image by correcting the captured image; a step ofevaluating the coverage based on the evaluation image; and a step ofoutputting a result of the evaluation carried out in the step ofevaluating the coverage. In the step of generating the evaluation image,an evaluation region is extracted from the captured image based on thesize of a dent region included in the captured image, and the evaluationimage is generated based on the evaluation region. The dent region is animage of a dent formed on the evaluation target.

An evaluation program according to further another aspect of the presentdisclosure is a program configured to cause a computer to execute: astep of acquiring a captured image of an evaluation target; a step ofgenerating an evaluation image by correcting the captured image; a stepof evaluating coverage of the evaluation target based on the evaluationimage; and a step of outputting a result of the evaluation carried outin the step of evaluating the coverage. In the step of generating theevaluation image, an evaluation region is extracted from the capturedimage based on the size of a dent region included in the captured image,and the evaluation image is generated based on the evaluation region.The dent region is an image of a dent formed on the evaluation target.

A recording medium according to further another aspect of the presentdisclosure is a computer-readable recording medium recording anevaluation program configured to cause a computer to execute: a step ofacquiring a captured image of an evaluation target; a step of generatingan evaluation image by correcting the captured image; a step ofevaluating coverage of the evaluation target based on the evaluationimage; and a step of outputting a result of the evaluation carried outin the step of evaluating the coverage. In the step of generating theevaluation image, an evaluation region is extracted from the capturedimage based on the size of a dent region included in the captured image,and the evaluation image is generated based on the evaluation region.The dent region is an image of a dent formed on the evaluation target.

In the evaluation system, the evaluation device, the evaluation method,the evaluation program, and the recording medium, the evaluation regionis extracted from the captured image of the evaluation target, and theevaluation image is generated based on the evaluation region. Then,coverage is evaluated based on the evaluation image, and the evaluationresult is output. The evaluation region is extracted from the capturedimage based on the size of the dent region, which is the image of thedent formed on the evaluation target. Therefore, for example, if thedent region is large, the evaluation region can be extracted so that thearea of the evaluation region becomes large. By virtue of this, thecoverage is evaluated for the range which corresponds to the size of thedent region. As a result, the evaluation accuracy of the coverage can beimproved.

The correction unit may extract the evaluation region from the capturedimage so that the larger the size of the dent region is, the evaluationregion becomes larger. In such a case, errors in the coverage caused bythe size of dents can be reduced. As a result, the evaluation accuracyof the coverage can be further improved.

The correction unit may set the size of the evaluation region bymultiplying the size of the dent region by a constant determined inadvance to extract the evaluation region from the captured image. Insuch a case, the influence of a single dent on the coverage can bereduced since the range (area) of the evaluation target can besufficiently increased with respect to the size of the dent region. As aresult, the evaluation accuracy of the coverage can be further improved.

The correction unit may expand or contract the evaluation region so asto adjust the size of the dent region to a predetermined size. In such acase, the evaluation by a neural network can be appropriately carriedout.

The correction unit may correct the color of the evaluation region basedon the color of the reference region included in the captured image. Thereference region may be an image of a reference colored by a specificcolor. Even if the evaluation target is the same, the color tone of thecaptured image sometimes changes depending on the color tone of a lightsource used to capture the image. Also, even if the evaluation target isthe same, the brightness of the captured image is sometimes differentdepending on the irradiation amount of light. According to the abovedescribed configuration, if the color of the reference region isdifferent from the specific color, the color of the captured image isconceivably affected by light. Therefore, the influence of light can bereduced, for example, by correcting the color of the evaluation regionso that the color of the reference region becomes the specific color(for example, the original color). By virtue of this, the evaluationaccuracy of the coverage can be further improved.

The correction unit may remove specular reflection from the evaluationregion. If the evaluation target is irradiated with intense light,specular reflection sometimes occur; and, if an image of the evaluationtarget is captured in that state, over exposure sometimes occur in thecaptured image. In the region in which the over exposure occurs, colorinformation is lost. Therefore, the color information can be restored byremoving the specular reflection (over exposure). By virtue of this, theevaluation accuracy of the coverage can be further improved.

The evaluation unit may evaluate coverage by using a neural network. Insuch a case, by causing the neural network to learn, the evaluationaccuracy of the coverage can be further improved.

Advantageous Effects of Invention

According to the aspects and embodiments of the present disclosure, theevaluation accuracy of coverage can be improved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a first embodiment.

FIG. 2 is a hardware configuration diagram of a user terminal shown inFIG. 1.

FIG. 3 is a hardware configuration diagram of the evaluation deviceshown in FIG. 1.

FIG. 4 is a sequence diagram showing an evaluation method carried out bythe evaluation system shown in FIG. 1.

FIG. 5 is a flowchart showing details of correction processing shown inFIG. 4.

FIGS. 6(a) to 6(f) are diagrams showing examples of a marker.

FIG. 7 is a diagram for describing distortion correction.

FIGS. 8(a) and 8(b) are diagrams for describing extraction of anevaluation region.

FIGS. 9(a) and 9(b) are diagrams for describing color correction.

FIG. 10 is a diagram showing an example of the neural network.

FIG. 11 is a diagram showing an example of an evaluation result.

FIGS. 12(a) and 12(b) are diagrams showing display examples of theevaluation result.

FIGS. 13(a) and 13(b) are diagrams showing correction example of theevaluation result.

FIG. 14 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a second embodiment.

FIG. 15 is a sequence diagram showing an evaluation method carried outby the evaluation system shown in FIG. 14.

FIG. 16 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a third embodiment.

FIG. 17 is a flowchart showing an evaluation method carried out by theevaluation system shown in FIG. 16.

FIGS. 18(a) to 18(d) are diagrams showing modification examples of amarker.

FIG. 19 is a diagram for describing a modification example of anextraction method of the evaluation region.

FIG. 20 is a diagram for describing a modification example of anextraction method of the evaluation region.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the drawings. Note that the same elements inthe description of the drawings are denoted by the same reference signs,and redundant descriptions will be omitted.

First Embodiment

FIG. 1 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a first embodiment.The evaluation system 1 shown in FIG. 1 is a system which evaluates thecoverage of an evaluation target. Examples of the evaluation targetinclude Almen strips, gears, and springs. The coverage is the rate ofthe area in which dents are formed by shots with respect to the totalsurface area, which is the target of measurement.

The evaluation system 1 includes a single or a plurality of userterminal(s) 10 and an evaluation device 20. The user terminal 10 and theevaluation device 20 are connected by a network NW so that they cancommunicate with each other. The network NW may either be wired orwireless. Examples of the network NW include the Internet, a mobilecommunication network, and a wide area network (WAN).

The user terminal 10 is a terminal device used by a user. The userterminal 10 generates a captured image of an evaluation target bycapturing an image of the evaluation target and transmits the capturedimage to the evaluation device 20. The user terminal 10 receives anevaluation result from the evaluation device 20 and outputs theevaluation result to the user. The user terminal 10 may be applied to amobile terminal in which an image capture device is built or may beapplied to a device which can communicate with an image capture device.The mobile terminal in which the image capture device is built is usedas the user terminal 10 to describe the present embodiment. Examples ofthe mobile terminal include a smartphone, a tablet terminal, and alaptop personal computer (PC).

FIG. 2 is a hardware configuration diagram of the user terminal shown inFIG. 1. As shown in FIG. 2, the user terminal 10 can be physicallyformed as a computer including hardware such as a single or a pluralityof processor(s) 101, a main storage device 102, an auxiliary storagedevice 103, a communication device 104, an input device 105, an outputdevice 106, and an image capture device 107. A processor having a highprocessing speed is used as the processor 101. Examples of the processor101 include a graphics processing unit (GPU) and a central processingunit (CPU). The main storage device 102 includes, for example, a randomaccess memory (RAM) and a read only memory (ROM). Examples of theauxiliary storage device 103 include a semiconductor memory and a harddisk device.

The communication device 104 is a device which transmits/receives datato/from other devices via the network NW. Examples of the communicationdevice 104 include a network card. Encryption may be used for thetransmission and reception of data via the network NW. In other words,the communication device 104 may encrypt data and transmit the encrypteddata to other devices. The communication device 104 may receiveencrypted data from other devices and decrypt the encrypted data. Commonkey cryptography such as triple data encryption standard (DES) andRijndael or public key cryptography such as RSA and ElGamal can be usedin the encryption.

The input device 105 is a device which is used when the user is tooperate the user terminal 10. Examples of the input device 105 include atouch screen, a keyboard, and a mouse. The output device 106 is a devicewhich outputs various information to the user of the user terminal 10.Examples of the output device 106 include a display, a speaker, and avibrator.

The image capture device 107 is a device for capturing images (imaging).The image capture device 107 is, for example, a camera module.Specifically, the image capture device 107 includes: a plurality ofoptical-system parts such as lenses and image capture elements, aplurality of control-system circuits which drive and control them, and asignal-processing-system circuit unit which converts the electricsignals, which represent captured images generated by the image captureelements, to image signals, which are digital signals.

The functions of the user terminal 10 shown in FIG. 1 are realized bycausing the hardware of the main storage device 102, etc. to read asingle or a plurality of predetermined computer program(s) so as tooperate the hardware under control of the single or plurality ofprocessor(s) 101 and by reading and writing data in the main storagedevice 102 and the auxiliary storage device 103.

The user terminal 10 includes, in terms of function, an imageacquisition unit 11, a correction unit 13, a transmission unit 14, areception unit 15, an output unit 16, and a corrected informationacquisition unit 17.

The image acquisition unit 11 is a part for acquiring a captured imageincluding an evaluation target. The image acquisition unit 11 isrealized, for example, by the image capture device 107. The capturedimage may be a still image or a moving image. The captured image is, forexample, acquired as image data, which shows pixel values of pixels(pixels). However, for explanatory convenience, the captured image isdescribed as a captured image. If the user terminal 10 does not have theimage capture device 107, the image acquisition unit 11 acquires thecaptured image, for example, by receiving the captured image, which hasbeen captured by another device (for example, a terminal having a camerafunction), from the other device. For example, in a case in which theimage acquisition unit 11 receives the captured image from the otherdevice via the network NW, the part that processes reception of thecaptured image (for example, the communication device 104 of FIG. 2)functions as the image acquisition unit 11. The image acquisition unit11 outputs the captured image to the correction unit 13.

The correction unit 13 is a part for generating an evaluation image bycorrecting the captured image. The correction unit 13 extracts anevaluation region from the captured image and generates the evaluationimage based on the evaluation region. The evaluation region isdetermined depending on the size of a dent region, which is an image ofa dent included in the captured image. The correction unit 13 performs,for example, size correction, distortion correction, color correction,specular reflection removal, noise removal, and blur correction on thecaptured image. Details of each correction processing will be describedlater. The correction unit 13 outputs the evaluation image to thetransmission unit 14.

The transmission unit 14 is a part for transmitting the evaluation imageto the evaluation device 20. The transmission unit 14 transmits theevaluation image to the evaluation device 20 via the network NW. Thetransmission unit 14 further transmits corrected information, which isacquired by the corrected information acquisition unit 17, to theevaluation device 20. The transmission unit 14 is realized, for example,by the communication device 104. The reception unit 15 is a part forreceiving an evaluation result from the evaluation device 20. Thereception unit 15 receives the evaluation result from the evaluationdevice 20 via the network NW. The reception unit 15 is realized, forexample, by the communication device 104.

The output unit 16 is a part for outputting the evaluation result. Theoutput unit 16 is realized, for example, by the output device 106. In acase in which the evaluation result is output by an output device suchas a display of another device, the output unit 16, for example,transmits the evaluation result to the other device via the network NW.In that case, the part which processes the transmission of theevaluation result (for example, the communication device 104 of FIG. 2)functions as the output unit 16.

The corrected information acquisition unit 17 is a part for acquiringthe corrected information of the evaluation result. For example, theuser sometimes checks the evaluation result, which is output by theoutput unit 16, and then corrects the evaluation result by using theinput device 105. In that case, the corrected information acquisitionunit 17 acquires the corrected evaluation result as correctedinformation. The corrected information acquisition unit 17 outputs thecorrected information to the transmission unit 14.

The evaluation device 20 is a device which evaluates the coverage of theevaluation target by using the captured image (evaluation image) of theevaluation target. The evaluation device 20 includes, for example, aninformation processing device (server device) such as a computer.

FIG. 3 is a hardware configuration diagram of the evaluation deviceshown in FIG. 1. As shown in FIG. 3, the evaluation device 20 can bephysically formed as a computer including hardware such as a single or aplurality of processor(s) 201, a main storage device 202, an auxiliarystorage device 203, and a communication device 204. A processor having ahigh processing speed is used as the processor 201. Examples of theprocessor 201 include a GPU and a CPU. The main storage device 202 isformed by, for example, a RAM and a ROM. Examples of the auxiliarystorage device 203 include a semiconductor memory and a hard diskdevice.

The communication device 204 is a device which transmits/receives datato/from other devices via the network NW. Examples of the communicationdevice 204 include a network card. Encryption may be used for thetransmission and reception of data via the network NW. In other words,the communication device 204 may encrypt data and transmit the encrypteddata to other devices. The communication device 204 may receiveencrypted data from other devices and decrypt the encrypted data. Commonkey cryptography such as triple DES and Rijndael or public keycryptography such as RSA and ElGamal can be used in the encryption.

The communication device 204 may carry out user authentication todetermine whether the user of the user terminal 10 is a valid user or aninvalid user. In that case, the evaluation device 20 may be configuredto carry out coverage evaluation if the user is a valid user and not tocarry out coverage evaluation if the user is an invalid user. Forexample, a user identifier (ID) and a password registered in advance areused in the user authentication. A one-time pad (one-time password) maybe used in the user authentication.

The functions of the evaluation device 20 shown in FIG. 1 are realizedby causing the hardware of the main storage device 202, etc. to read asingle or a plurality of predetermined computer program(s) so as tooperate the hardware under control of the single or plurality ofprocessor(s) 201 and reading and writing data by the main storage device202 and the auxiliary storage device 203.

The evaluation device 20 includes, in terms of function, a receptionunit 21, an evaluation unit 22, and a transmission unit 23.

The reception unit 21 is a part for receiving the evaluation image fromthe user terminal 10. The reception unit 21 receives the evaluationimage from the user terminal 10 via the network NW. The reception unit21 further receives the corrected information from the user terminal 10.The reception unit 21 is realized, for example, by the communicationdevice 204. The reception unit 21 outputs the evaluation image and thecorrected information to the evaluation unit 22.

The evaluation unit 22 is a part for evaluating the coverage of theevaluation target based on the evaluation image. The evaluation unit 22evaluates the coverage of the evaluation target by using a neuralnetwork. The neural network may be a convolutional neural network(Convolutional Neural Network: CNN) or may be a recurrent neural network(Recurrent Neural Network: RNN). The evaluation unit 22 outputs theevaluation result to the transmission unit 23.

The transmission unit 23 is a part for transmitting the evaluationresult to the user terminal 10. The transmission unit 23 transmits theevaluation result to the user terminal 10 via the network NW. Thetransmission unit 23 is realized, for example, by the communicationdevice 204. Note that, since the transmission unit 23 outputs(transmits) the evaluation result to the user terminal 10, thetransmission unit 23 may be considered as an output unit.

Next, an evaluation method carried out by the evaluation system 1 willbe described with reference to FIG. 4 to FIG. 13(b). FIG. 4 is asequence diagram showing the evaluation method carried out by theevaluation system shown in FIG. 1. FIG. 5 is a flowchart showing detailsof the correction processing shown in FIG. 4. FIGS. 6(a) to 6(f) arediagrams showing examples of a marker. FIG. 7 is a diagram fordescribing distortion correction. FIGS. 8(a) and 8(b) are diagrams fordescribing extraction of the evaluation region. FIGS. 9(a) and 9(b) arediagrams for describing color correction. FIG. 10 is a diagram forshowing an example of the neural network. FIG. 11 is a diagram showingan example of the evaluation result. FIGS. 12(a) and 12(b) are diagramsshowing display examples of the evaluation result. FIGS. 13(a) and 13(b)are diagrams showing correction example of the evaluation result.

The series of processes of the evaluation method shown in FIG. 4 isstarted, for example, when the user of the user terminal 10 captures animage of an evaluation target by using the image capture device 107.First, the image acquisition unit 11 acquires a captured image of theevaluation target (step S01). For example, the image acquisition unit 11acquires, as the captured image, an image of the evaluation targetgenerated by the image capture device 107. Then, the image acquisitionunit 11 outputs the acquired captured image to the correction unit 13.

Note that, before the captured image of the evaluation target isacquired, a marker MK may be attached to the evaluation target. Themarker MK is used for correcting the captured image in later-describedimage processing. The marker MK has a shape with which the direction ofthe marker MK can be specified. The marker MK is asymmetric at least ineither one of a top-bottom direction and a width direction.Specifically, as shown in FIGS. 6(a) to 6(f), the marker MK includes awhite colored region Rw and a black colored region Rb. In order tofacilitate the later-described image processing, the marker MK has atetragonal edge F1. The edge F1 is an edge of the region Rb. As shown inFIGS. 6(b) to 6(f), the marker MK may be surrounded by a frame F2, and agap Rgap may be provided between the frame F2 and the region Rb.

The marker MK is drawn on a sheet-like member. For example, the user ofthe user terminal 10 directly pastes the sheet-like member, whichincludes the marker MK, onto the evaluation target. The user may pastethe sheet-like member, which includes the marker MK, onto the evaluationtarget by using an unmanned aerial vehicle (UAV) or an extensible rod.

Note that the marker MK is only required to include two or moredifferently colored regions. For example, the color of the region Rw isnot required to be white, but may be, for example, gray. The color ofthe region Rb is not required to be black, but may be a color havingsaturation. In the present embodiment, the marker MK shown in FIG. 6(a)is used.

Subsequently, the correction unit 13 corrects the captured image (stepS02). As shown in FIG. 5, in the correction processing of step S02,first, the correction unit 13 carries out distortion correction in orderto correct the distortion of the captured image (step S21). The capturedimage is sometimes distorted compared with an image which is obtained bycapturing an image of the evaluation target from a front side. Forexample, if the image capture device 107 is a depth camera, thedistances between the image capture device 107 and positions of theevaluation target are obtained. In that case, the correction unit 13carries out distortion correction by converting the captured image tothe image, which is obtained by capturing an image of the evaluationtarget from the front side, based on the distances between the imagecapture device 107 and the positions of the evaluation target. If theevaluation target is a structure having an aspect such as a spring, thecorrection unit 13 may further carry out aspect correction as distortioncorrection.

The correction unit 13 may carry out the distortion correction by usingthe marker MK. The captured image of the evaluation target with themarker MK includes a marker region Rm, which is an image (image region)of the marker MK. In that case, first, the correction unit 13 extractsthe marker region Rm from the captured image. The correction unit 13extracts the marker region Rm, for example, by performing objectdetection processing or edge detection processing on the captured image.If the marker MK has a simple shape, edge detection processing sometimeshas higher detection accuracy and higher processing speed than objectdetection processing, and, therefore, edge detection processing may beused.

Then, the correction unit 13 checks whether the extracted marker regionRm is the image of the marker MK or not. The correction unit 13, forexample, carries out histogram averaging processing with respect to themarker region Rm and then carries out binarization processing withrespect to the marker region Rm. Then, the correction unit 13 comparesthe binarized marker region Rm with the marker MK and, if they match,determines that the marker region Rm is the image of the marker MK. As aresult, the vertex coordinates of the marker MK in the captured imageare acquired. If they do not match, the correction unit 13 determinesthat the marker region Rm is not the image of the marker MK and extractsthe marker region Rm again.

Then, the correction unit 13 calculates the direction of the marker MKin the captured image by using the marker region Rm. Since the marker MKis asymmetric at least in either one of the top-bottom direction and thewidth direction, the direction of the marker MK in the captured imagecan be calculated. Then, as shown in FIG. 7, the correction unit 13subjects the captured image to projection transformation so that theoriginal shape of the marker MK is restored from the vertex coordinatesand direction of the marker MK in the captured image, thereby convertingthe captured image to the image, which is obtained by capturing an imageof the evaluation target from the front side. Specifically, thecorrection unit 13 uses a vertex Pm1 as an origin, the direction fromthe vertex Pm1 toward a vertex Pmt as an X1 axis direction, and thedirection from the vertex Pm1 toward a vertex Pm4 as a Y1 axisdirection. Then, the correction unit 13 converts an X1-Y1 coordinatesystem to an X-Y orthogonal coordinate system, thereby restoring theshape of the marker MK. As a result, the distortion correction iscarried out.

Subsequently, the correction unit 13 extracts an evaluation region Refrom the captured image (step S22). Since a single shot peening iscarried out with the shot media having the same size, the sizes of dentsare similar to one another. However, the types of shot media used inshot peening include, for example, shot media having diameters (graindiameters) of about 0.1 mm to 1 mm. Therefore, the size of the shotmedia used in a shot peening is sometimes different from the size of theshot media used in another shot peening. If coverage is evaluated forthese shot media by using the same area, the influence on the coverageevaluation caused by a single dent is different depending on the size(diameter) of the shot media. Therefore, as shown in FIGS. 8(a) and8(b), the correction unit 13 extracts the evaluation region Re from thecaptured image G based on the size of a dent region De included in thecaptured image G and generates an evaluation image based on theevaluation region Re. The dent region De is an image of a dent formed onthe evaluation target.

As the size of the dent region De, for example, the average size (forexample, average diameter) of a plurality of dent regions De included inthe captured image G is used. The correction unit 13 detects theplurality of dent regions De included in the captured image U, forexample, by object detection. Then, the correction unit 13 calculatesthe average size (for example, average diameter) of the plurality ofdent regions De included in the captured image G and extracts theevaluation region Re from the captured image G so that the larger theaverage size of the dent region De is, the evaluation region Re becomeslarger. Specifically, the correction unit 13 sets the size of theevaluation region Re by multiplying the average size (average diameter)of the dent region De by a multiplying factor (for example, by 5 to 10)determined in advance. For example, the correction unit 13 extracts asquare region having a length of a side which is equal to themultiplication result as the evaluation region Re from the capturedimage.

Subsequently, the correction unit 13 corrects the size of the evaluationregion Re (step S23). The size of the evaluation region Re can bechanged depending on the size of the dent region De. Therefore, in thesize correction, the correction unit 13 carries outexpansion/contraction processing of the evaluation region Re so that thesize of the dent region De is adjusted to a predetermined size(reference grain diameter). As a result, the size of the evaluationregion Re is adjusted to a predetermined evaluation size. The evaluationsize is the size of a reference image (teaching data) used in learningof a neural network NN.

In the expansion/contraction processing, first, the correction unit 13compares the size (average diameter) of the dent region De with thereference grain diameter and determines which one of expansionprocessing and contraction processing is to be carried out. If theaverage diameter of the dent regions De is smaller than the referencegrain diameter, the correction unit 13 carries out expansion processing.If the average diameter of the dent regions De is larger than thereference grain diameter, the correction unit 13 carries out contractionprocessing. In other words, the correction unit 13 adjusts the size ofthe evaluation image to the evaluation size by expanding or contractingthe evaluation region Re. In the expansion processing, for example,bilinear interpolation is used. In the contraction processing, forexample, the average pixel method is used. Other expansion/contractionalgorithms may be used in the expansion processing and the contractionprocessing, and it is desired to maintain the state of the image evenafter the expansion/contraction processing.

Subsequently, the correction unit 13 carries out color correction of theevaluation region Re (step S24). Even for the same evaluation target,the brightness of the image may change depending on an image capturingenvironment. Moreover, if the color of a light source used to capture animage is different, the color of the image may also be different. Inorder to reduce the influence of the image capturing environment, colorcorrection is carried out. The correction unit 13 corrects the colors ofthe evaluation region Re based on the color of a reference regionincluded in the captured image. The reference region is an image (imageregion) of a reference colored by a specific color.

As shown in FIG. 9(a), the region Rw of the marker MK can be used as thereference. In such a case, the color of the region Rw of the marker MKis measured in advance by a color meter or the like, and reference valueindicating the measured color are stored in a memory, which is not shownin the drawings. As the value indicating the color, for example, RGBvalue and HSV value are used. As shown in FIG. 9 (b), the correctionunit 13 acquires the values of the color of the region Rw of the markerregion Rm included in the captured image (evaluation region Re),compares the acquired values with reference value, and carries out colorcorrection so that the differences therebetween are reduced (forexample, to zero). As the color correction, for example, gammacorrection is used. Differences may be added to respective pixel values(offset processing) as color correction.

The marker MK is not required to be used as the reference. In such acase, like the case in which the marker MK is used, color correction ofthe evaluation region Re may be carried out by using a sample (forexample, a gray board), which has a color measured in advance, as areference and capturing an image thereof with the evaluation target. Thecorrection unit 13 may carry out color correction based on the grayworld assumption.

Subsequently, the correction unit 13 removes specular reflection fromthe evaluation region Re (step S25). Specular reflection is sometimescaused when the evaluation target has metallic luster. Specularreflection is sometimes caused depending on the state of coating of theevaluation target. In an image, the part at which specular reflectionhas occurred usually appears to be intense white. In other words, thepart at which specular reflection has occurred causes over exposure inan image. After the color correction, the part at which specularreflection has occurred can be detected as a white part. Therefore, thecorrection unit 13 removes the specular reflection by using the image(evaluation region Re) after the color correction.

Therefore, the correction unit 13 specifies a specular reflection partbased on the pixel values of the pixels included in the evaluationregion Re. For example, if all of the pixel values of RGB are largerthan predetermined threshold values, the correction unit 13 determinesthat the pixel is part of the specular reflection part. The correctionunit 13 may specify the specular reflection part by converting the pixelvalues to HSV and carrying out similar threshold value processing withrespect to brightness (V) or both of brightness (V) and saturation (S).

Then, the correction unit 13 removes the specular reflection from thespecular reflection part to restore the original image information(pixel values). The correction unit 13 automatically interpolates(restores) the image information of the specular reflection part withthe information of the image which is in the vicinity of the specularreflection part, for example, by a method using the Navier-Stokesequations and the fast marching method of Alexandru Telea. Thecorrection unit 13 may restore the image information of the specularreflection part by learning images, which have various coverage values,in advance by machine learning. For example, generative adversarialnetwork (GAN) is used for the machine learning. Note that the correctionunit 13 may restore image information on a region expanded from an outeredge of the specular reflection part (in other words, a region whichincludes the specular reflection part and is larger than the specularreflection part).

Subsequently, the correction unit 13 removes noise from the evaluationregion Re (step S26). The correction unit 13 removes noise from theevaluation region Re, for example, by using denoise filters (denoisefunction) such as a Gaussian filter and a low-pass filter.

Subsequently, the correction unit 13 carries out blur correction of theevaluation region Re (step S27). When the user captures an image byusing the user terminal 10, blurs such as blurs caused by handinstability sometimes occur. The correction unit 13 carries out blurcorrection of the image, for example, by using a Wiener filter and theblind deconvolution algorithm.

Note that the correction processing of FIG. 5 is an example, and thecorrection processing carried out by the correction unit 13 is notlimited thereto. Part or all of steps S21 and S23 to S27 may be omitted.Steps S21 to S27 may be carried out in an arbitrary order. In a case inwhich specular reflection removal is carried out after color correctionas described above, the accuracy to specify specular reflection parts isimproved since the specular reflection parts in an image appears to beintense white.

As shown in FIG. 7, the correction unit 13 may consider that the markerregion Rm is formed of a plurality of blocks arranged like a matrix andobtain the vertex coordinates of each of blocks by using the coordinatesof four vertices of the marker region Rm (marker MK). By virtue of this,the correction unit 13 can handle the marker region Rm as a plurality ofdivided blocks. For example, the correction unit 13 determines whetherthe marker region Rm is the image of the marker MK or not by using theblocks. Also, the correction unit 13 may use any of the blocks as thereference region which is used for color correction. Furthermore, thecorrection unit 13 may calculate the degree of distortion in thecaptured image from the coordinates of the blocks to carry outcalibration of the image capture device 107.

Subsequently, the correction unit 13 outputs the captured image, whichhas been corrected by the correction processing of step S02, to thetransmission unit 14 as the evaluation image, and the transmission unit14 transmits the evaluation image to the evaluation device 20 via thenetwork NW (step S03). In this process, the transmission unit 14transmits, to the evaluation device 20, the evaluation image togetherwith a terminal ID with which the user terminal 10 can be uniquelyidentified. For example, an internet protocol (IP) address may be usedas the terminal ID. Then, the reception unit 21 receives the evaluationimage, which has been transmitted from the user terminal 10, and outputsthe evaluation image to the evaluation unit 22. If the evaluation imagelacks clarity, the correction unit 13 does not have to output theevaluation image to the transmission unit 14. As described above, thetransmission unit 14 may encrypt the evaluation image and transmit theencrypted evaluation image to the evaluation device 20. In such a case,the reception unit 21 receives the encrypted evaluation image from theuser terminal 10, decrypts the encrypted evaluation image, and outputsthe evaluation image to the evaluation unit 22.

Subsequently, the evaluation unit 22 evaluates the coverage of theevaluation target based on the evaluation image (step S04). In thisexample, the evaluation unit 22 evaluates the coverage of the evaluationtarget by using the neural network NN shown in FIG. 10. Note that, whenthe evaluation unit 22 receives the evaluation image, the evaluationunit 22 imparts an image ID, with which the evaluation image can beuniquely identified, to the evaluation image.

The neural network NN uses the evaluation image as an input and outputsthe match rate of each of categories. As the categories, the valuesgrouping the coverage in predetermined rate units can be used. Forexample, in a case in which the coverage is expressed by percentage, 0to 98% are categorized by a 10% unit. Note that examples of the standardabout the coverage include HS B2711 and SAE J2277. For example, in SAEJ2277, the upper limit of measurable coverage is 98% (full coverage).The categories are not limited to the 10% unit, but may be set by a 5%unit or may be set by a 1% unit.

As shown in FIG. 11, in the present embodiment, the values which groupthe coverage of 0 to 98% by the 10% unit are used as the categories. Inthis example, for explanatory convenience, a category “100%” is used.The match rate expresses the probability that the coverage of theevaluation target belongs to that category. This means that the higherthe match rate is, the possibility that the coverage of the evaluationtarget belongs to that category becomes higher.

The evaluation unit 22 may separate the evaluation image into one or aplurality of channels and use the image information (pixel values) ofeach channel as the input of the neural network NN. The evaluation unit22, for example, separates the evaluation image into components of acolor space. In a case in which an RGB color space is used as the colorspace, the evaluation unit 22 separates the evaluation image into thepixel values of R components, the pixel values of G components, and thepixel values of B components. In a case in which a HSV color space isused as the color space, the evaluation unit 22 separates the evaluationimage into the pixel values of H components, the pixel values of Scomponents, and the pixel values of V components. The evaluation unit 22may convert the evaluation image to grayscale image and use theconverted image as the input of the neural network NN.

As shown in FIG. 10, the neural network NN has an input layer L1, anintermediate layer L2, and an output layer L3. The input layer L1 ispositioned at the entrance of the neural network NN, and M input valuesx_(i) (i is an integer of 1 to M) are input to the input layer L1. Theinput layer L1 has a plurality of neurons 41. The neurons 41 areprovided to correspond to the input values x_(i), and the number of theneurons 41 is equal to the total number M of the input values x_(i). Inother words, the number of the neurons 41 is equal to the summation ofthe number of the pixels included in each channel of the evaluationimage. The i-th neuron 41 outputs the input value x_(i) to each ofneurons 421 of a first intermediate layer L21 of the intermediate layerL2. The input layer L1 includes a node 41 b. The node 41 b outputs abias value b_(j) (j is an integer of 1 to M1) to each of the neurons421.

The intermediate layer L2 is positioned between the input layer L1 andthe output layer L3. The intermediate layer L2 is also referred to as ahidden layer since it is hidden from the outside of the neural networkNN. The intermediate layer L2 includes one or a plurality of layers. Inthe example shown in FIG. 10, the intermediate layer L2 includes thefirst intermediate layer L21 and a second intermediate layer L22. Thefirst intermediate layer L21 has the M1 neurons 421. In such a case, thej-th neuron 421 obtains a calculated value z_(j) by further adding abias value b_(j) to the summation of the values, which are obtained byweighting each input value x_(i) by a weight coefficient w_(ij), asshown in Formula (1). Note that, if the neural network NN is aconvolutional neural network, the neuron 421 carries out, for example,convolution, calculation using an activation function, and poolingsequentially. In such a case, for example, the ReLU function is used asthe activation function.

$\begin{matrix}\lbrack {{Formula}\mspace{14mu} 1} \rbrack & \; \\{z_{j} = {{\sum\limits_{i = 1}^{M}{w_{ij} \times x_{i}}} + b_{j}}} & (1)\end{matrix}$

Then, the j-th neuron 421 outputs the calculated value z₁ to each ofneurons 422 of the second intermediate layer L22. The first intermediatelayer L21 includes a node 421 b. The node 421 b outputs a bias value toeach of the neurons 422. Thereafter, each of the neurons carries outcalculations like the neuron 421 and outputs a calculated value to eachof the neurons of a subsequent level. The neurons (in this case, theneurons 422) of the final level of the intermediate layer L2 output thecalculated values to each of neurons 43 of the output layer L3.

The output layer L3 is positioned at the exit of the neural network NNand outputs output values y_(k) (k is an integer of 1 to N). The outputvalue y_(k) is allocated to each category and is the value correspondingto the match rate of the category. The output layer L3 has the pluralityof neurons 43. The neurons 43 are provided to correspond to the outputvalues y_(k), and the number of the neurons 43 is equal to the totalnumber N of the output values y_(k). In other words, the number of theneurons 43 is equal to the number of the categories indicating thecoverage. Each of the neurons 43 carries out calculations like theneuron 421 and calculates an activation function by using thecalculation result thereof as an argument, thereby obtaining the outputvalue y_(k). Examples of the activation function include: a softmaxfunction, a ReLU function, a hyperbolic function, a sigmoid function, anidentity function, and a step function. In the present embodiment, asoftmax function is used. Therefore, the output values y_(k) arenormalized so that the sum of the N output values y_(k) becomes 1. Inother words, the match rate (%) can be obtained by multiplying theoutput value y_(k) by 100.

Subsequently, the evaluation unit 22 outputs, for example, the N outputvalues y_(k) together with the image ID of the evaluation image to thetransmission unit 23 as an evaluation result of the evaluation image.The sequence of the N output values y_(k) is determined in advance, andeach of the output values y_(k) is associated with any of the categoriesof the N categories. Note that the evaluation unit 22 may use thelargest output value among the N output values y_(k) together with acategory name or an index (corresponding to “number” shown in FIG. 11)corresponding to the output value as an evaluation result. In this case,the sequence of the output values corresponding to the match rates shownin FIG. 11 is output to the transmission unit 23 as the evaluationresult. In this case, the user terminal 10 can determine the manner ofoutput to the user.

Then, the transmission unit 23 transmits the evaluation result to theuser terminal 10 via the network NW (step S05). In this process, thetransmission unit 23 identifies the user terminal 10 of the transmissiondestination based on the terminal ID, which has been transmitted fromthe user terminal 10 together with the evaluation image, and transmitsthe evaluation result to the user terminal 10. Then, the reception unit15 receives the evaluation result, which has been transmitted from theevaluation device 20, and outputs the evaluation result to the outputunit 16. Note that, as described above, the transmission unit 23 mayencrypt the evaluation result and transmit the encrypted evaluationresult to the user terminal 10. In such a case, the reception unit 15receives the encrypted evaluation result from the evaluation device 20,decrypts the encrypted evaluation result, and outputs the evaluationresult to the output unit 16.

Subsequently, the output unit 16 generates output information, which isfor informing the user of the evaluation result, and outputs theevaluation result to the user based on the output information (stepS06). The output unit 16 displays, for example, the name of the category(coverage oo %) having the highest match rate and the match ratethereof. Also, the output unit 16 may, for example, calculate coverageby totaling the results of multiplication of the value of each categoryby the match rate thereof and display the calculation result thereof asan evaluation result. In the example of FIG. 11, the coverage is 45%(=40%×0.5+50%×0.5).

As shown in FIGS. 12(a) and 12(b), the output unit 16 may display theevaluation result of the coverage by using an arrow Pa in a graph. Theoutput unit 16 may display the evaluation result by texts. For example,the output unit 16 displays “Result: coverage 45%” or the like. Theoutput unit 16 may display the names of all the categories and the matchrates thereof by texts.

The output unit 16 may inform the user whether the shot peeningtreatment is successful or not by using the evaluation result. Theoutput unit 16 may output the evaluation result by sound or may outputthe evaluation result by vibrations. The form of output by the outputunit 16 may be set by the user.

Subsequently, the corrected information acquisition unit 17 determineswhether or not a correction operation of the evaluation result has beencarried out by the user. For example, the user operates so as to displaya screen for correcting the evaluation result by using the input device105 after checking the evaluation result, which has been output by theoutput unit 16.

For example, as shown in FIGS. 13(a) and 13(b), the user specifiescoverage on a graph by operating the input device 105 to move an arrowPa by using a pointer MP. In other words, the user determines coverageby visually checking the evaluation target, and the user moves the arrowPa so that the numerical value corresponding to the coverage determinedby the user is indicated.

A text box may be used to specify the coverage by the user. Objects suchas radio buttons, a drop down list, or sliders may be used for the userto select the category.

If the corrected information acquisition unit 17 determines that thecorrection operation has not been carried out, the series of processesof the evaluation method by the evaluation system 1 is finished. On theother hand, if it is determined that the correction operation has beencarried out by the input device 105, the corrected informationacquisition unit 17 acquires, as corrected information, the informationindicating a corrected category together with the image ID of theevaluation image to which the correction operation has been carried out(step S07).

Then, the corrected information acquisition unit 17 outputs thecorrected information to the transmission unit 14, and the transmissionunit 14 transmits the corrected information to the evaluation device 20via the network NW (step S08). Then, the reception unit 21 receives thecorrected information transmitted from the user terminal 10 and outputsthe corrected information to the evaluation unit 22. Note that, asdescribed above, the transmission unit 14 may encrypt the correctedinformation and transmit the encrypted corrected information to theevaluation device 20. In such a case, the reception unit 21 receives theencrypted corrected information from the user terminal 10, decrypts theencrypted corrected information, and outputs the corrected informationto the evaluation unit 22.

Subsequently, the evaluation unit 22 carries out learning based on thecorrected information (step S09). Specifically, the evaluation unit 22uses the set of the corrected category and the evaluation image asteaching data. The evaluation unit 22 may carry out learning of theneural network NN by any method of online learning, mini batch learning,and batch learning. The online learning is a method in which learning iscarried out by using new teaching data every time new teaching data isacquired. The mini batch learning is a method in which learning iscarried out by using one unit of teaching data, wherein a certain amountof teaching data serves as one unit. The batch learning is a method inwhich learning is carried out by using all teaching data. An algorithmsuch as back propagation is used in the learning. Note that the learningof the neural network NN means to update the weight coefficient and thebias value used in the neural network NN to more suitable values.

In the above described manner, the series of processes of the evaluationmethod by the evaluation system 1 is finished.

Note that the functional units of the user terminal 10 and theevaluation device 20 are realized by executing program modules, whichare for realizing the functions, by computers, which constitute the userterminal 10 and the evaluation device 20. An evaluation programincluding these program modules are provided, for example, by acomputer-readable recording medium such as a ROM or a semiconductormemory. The evaluation program may also be provided via a network asdata signals.

In the evaluation system 1, the evaluation device 20, the evaluationmethod, the evaluation program, and the recording medium describedabove, the evaluation region Re is extracted from the captured image ofthe evaluation target, and the evaluation image is generated based onthe evaluation region Re. Then, coverage is evaluated based on theevaluation image, and the evaluation result is output. The evaluationregion Re is extracted from the captured image based on the size of thedent region De, which is the image of the dent formed on the evaluationtarget. Specifically, the evaluation region Re is extracted from thecaptured image so that the larger the dent region De is, (the area of)the evaluation region Re becomes larger. By virtue of this, the coverageis evaluated for the range corresponding to the size of the dent regionDe. Therefore, the influence of a single dent on the coverage can bereduced. As a result, the evaluation accuracy of the coverage can beimproved.

More specifically, the size of the evaluation region Re is set bymultiplying the size of the dent region De (for example, averagediameter) by a constant, which is determined in advance. Therefore, theinfluence of a single dent on the coverage can be reduced since therange (area) of the evaluation region Re can be sufficiently increasedwith respect to the size of the dent region De. As a result, theevaluation accuracy of the coverage can be improved.

The evaluation region Re is expanded or contracted so as to adjust thesize of the dent region De to a predetermined size (for example,reference grain diameter). Therefore, the evaluation by the neuralnetwork NN can be appropriately carried out. Moreover, the evaluationaccuracy of the coverage can be improved since the coverage can beevaluated by a common standard with respect to the shot media havingmutually different grain diameters.

Even if the evaluation target is the same, the color tone of thecaptured image sometimes changes depending on the color tone of a lightsource used to capture the image. Also, even if the evaluation target isthe same, the brightness of the captured image is sometimes differentdepending on the irradiation amount of light. Therefore, the color ofthe evaluation region Re is corrected based on the color of thereference region (for example, the region Rw in the marker region Rm)included in the captured image. If the color of the region Rw in themarker region Rm is different from the color (white) of the region Rw inthe marker MK, the color of the captured image is considered to beaffected by the light. Therefore, the color of the evaluation region Reis corrected so that the color of the region Rw in the marker region Rmbecomes the color of the region Rw in the marker MK. By virtue of this,the influence of light can be reduced. As a result, the evaluationaccuracy of the coverage can be further improved.

If the evaluation target is irradiated with intense light, specularreflection sometimes occur; and, if an image of the evaluation target iscaptured in that state, over exposure sometimes occur in the capturedimage. In the region in which the over exposure occurs, colorinformation is lost. Therefore, the color information can be restored byremoving the specular reflection (over exposure) from the evaluationregion Re. By virtue of this, the evaluation accuracy of the coveragecan be further improved.

The coverage is evaluated by using the neural network NN. The patterngenerated on the surface of the evaluation target by the shot peeningtreatment is irregular. Therefore, it is difficult to specify theposition and state of the irregular-shaped object by a general objectdetection. Moreover, pattern recognition is not suitable for recognizingthe countlessly existing patterns. On the other hand, by causing theneural network NN to learn, the coverage can be evaluated, and theevaluation accuracy of the coverage can be further improved.

Second Embodiment

FIG. 14 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a second embodiment.The evaluation system 1A shown in FIG. 14 is different from theevaluation system 1 mainly in a point that the evaluation system 1Aincludes a user terminal 10A instead of the user terminal 10 and a pointthat the evaluation system 1A includes an evaluation device 20A insteadof the evaluation device 20.

The user terminal 10A is different from the user terminal 10 mainly in apoint that the user terminal 10A does not include the correction unit 13and a point that the user terminal 10A transmits a captured image to theevaluation device 20A instead of the evaluation image. Note that, in theuser terminal 10A, the image acquisition unit 11 outputs a capturedimage to the transmission unit 14. The transmission unit 14 transmitsthe captured image to the evaluation device 20A.

The evaluation device 20A is different from the evaluation device 20mainly in a point that the evaluation device 20A receives the capturedimage from the user terminal 10A instead of the evaluation image and apoint that the evaluation device 20A further includes a correction unit24. The reception unit 21 receives the captured image from the userterminal 10A and outputs the captured image to the correction unit 24.Note that, the reception unit 21 can be considered as an imageacquisition unit since it acquires the captured image from the userterminal 10A. The correction unit 24 has a function similar to that ofthe correction unit 13. More specifically, the correction unit 24extracts an evaluation region from the captured image and generates theevaluation image based on the evaluation region. Then, the correctionunit 24 outputs the evaluation image to the evaluation unit 22.

Next, with reference to FIG. 15, an evaluation method carried out by theevaluation system 1A will be described. FIG. 15 is a sequence diagramshowing the evaluation method carried out by the evaluation system shownin FIG. 14. First, the image acquisition unit 11 acquires a capturedimage of an evaluation target (step S31). For example, as well as stepS01, the image acquisition unit 11 acquires, as the captured image, animage of the evaluation target generated by the image capture device107.

Then, the image acquisition unit 11 outputs the acquired captured imageto the transmission unit 14, and the transmission unit 14 transmits thecaptured image to the evaluation device 20A via the network NW (stepS32). In this process, the transmission unit 14 transmits, to theevaluation device 20A, the captured image together with a terminal IDwith which the user terminal 10A can be uniquely identified. Then, thereception unit 21 receives the captured image, which has beentransmitted from the user terminal 10A, and outputs the captured imageto the correction unit 24. Note that, as described above, thetransmission unit 14 may encrypt the captured image and transmit theencrypted captured image to the evaluation device 20A. In such a case,the reception unit 21 receives the encrypted captured image from theuser terminal 10A, decrypts the encrypted captured image, and outputsthe captured image to the correction unit 24.

Subsequently, the correction unit 24 corrects the captured image (stepS33). Since the processing of step S33 is similar to the processing ofstep S02, detailed descriptions thereof will be omitted. The correctionunit 24 outputs the captured image, which has been corrected by thecorrection processing of step S33, to the evaluation unit 22 as anevaluation image. Thereafter, the processing of step S34 to step S39 issimilar to the processing of step S04 to step S09. Therefore, detaileddescriptions thereof will be omitted. In the above described manner, theseries of processes of the evaluation method by the evaluation system 1Ais finished.

Note that the functional units of the user terminal 10A and theevaluation device 20A are realized by executing program modules, whichare for realizing the functions, by computers, which constitute the userterminal 10A and the evaluation device 20A. An evaluation programincluding these program modules are provided, for example, by acomputer-readable recording medium such as a ROM or a semiconductormemory. The evaluation program may also be provided via a network asdata signals.

Also by the evaluation system 1A, the evaluation device 20A, theevaluation method, the evaluation program, and the recording mediumaccording to the second embodiment, the effects similar to those of theevaluation system 1, the evaluation device 20, the evaluation method,the evaluation program, and the recording medium according to the firstembodiment are exerted. Moreover, according to the evaluation system 1A,the evaluation device 20A, the evaluation method, the evaluationprogram, and the recording medium according to the second embodiment,the processing load of the user terminal 10A can be reduced since theuser terminal 10A does not include the correction unit 13.

Third Embodiment

FIG. 16 is a configuration diagram schematically showing an evaluationsystem including an evaluation device according to a third embodiment.The evaluation system 1B shown in FIG. 16 is different from theevaluation system 1 mainly in a point that the evaluation system 1Bincludes a user terminal 10B instead of the user terminal 10 and a pointthat the evaluation system 1B does not include the evaluation device 20.The user terminal 10B is different from the user terminal 10 mainly in apoint that the user terminal 10B further includes an evaluation unit 18and a point that the user terminal 10B does not include the transmissionunit 14 and the reception unit 15. In this case, the user terminal 10Bis also an evaluation device of a standalone type.

Note that, in the user terminal 10B, the correction unit 13 outputs theevaluation image to the evaluation unit 18. The corrected informationacquisition unit 17 outputs the corrected information to the evaluationunit 18. The evaluation unit 18 has a function similar to that of theevaluation unit 22. More specifically, the evaluation unit 18 evaluatesthe coverage of the evaluation target based on the evaluation image.Then, the evaluation unit 18 outputs the evaluation result to the outputunit 16.

Next, with reference to FIG. 17, an evaluation method carried out by theevaluation system 1B (user terminal 10B) will be described. FIG. 17 is aflowchart showing the evaluation method carried out by the evaluationsystem shown in FIG. 16.

First, as well as step S01, the image acquisition unit 11 acquires acaptured image of an evaluation target (step S41). Then, the imageacquisition unit 11 outputs the captured image to the correction unit13. Subsequently, the correction unit 13 corrects the captured image(step S42). Since the processing of step S42 is similar to theprocessing of step S02, detailed descriptions thereof will be omitted.Then, the correction unit 13 outputs the captured image, which has beencorrected by the correction processing of step S42, to the evaluationunit 18 as an evaluation image.

Subsequently, the evaluation unit 18 evaluates the coverage of theevaluation target based on the evaluation image (step S43). Since theprocessing of step S43 is similar to the processing of step S04,detailed descriptions thereof will be omitted. Then, the evaluation unit18 outputs the evaluation result to the output unit 16. Subsequently,the output unit 16 generates output information, which is for informingthe user of the evaluation result, and outputs the evaluation result tothe user based on the output information (step S44). Since theprocessing of step S44 is similar to the processing of step S06,detailed descriptions thereof will be omitted.

Subsequently, the corrected information acquisition unit 17 determineswhether or not a correction operation of the evaluation result has beencarried out by the user (step S45). If the corrected informationacquisition unit 17 determines that the correction operation has notbeen carried out (step S45: NO), the series of processes of theevaluation method by the evaluation system 1B are finished. On the otherhand, if it is determined that the correction operation has been carriedout (step S45: YES), the corrected information acquisition unit 17acquires, as corrected information, the information indicating acorrected category together with the image ID of the evaluation image towhich the correction operation has been carried out. Then, the correctedinformation acquisition unit 17 outputs the corrected information to theevaluation unit 18.

Subsequently, the evaluation unit 18 carries out learning based on thecorrected information (step S46). Since the processing of step S46 issimilar to the processing of step S09, detailed descriptions thereofwill be omitted. In the above described manner, the series of processesof the evaluation method by the evaluation system 1B are finished.

Note that the functional units of the user terminal 10B are realized byexecuting program modules, which are for realizing the functions, by acomputer, which constitutes the user terminal 10B. An evaluation programincluding these program modules are provided, for example, by acomputer-readable recording medium such as a ROM or a semiconductormemory. The evaluation program may also be provided via a network asdata signals.

Also by the evaluation system 1B, the user terminal 10B, the evaluationmethod, the evaluation program, and the recording medium according tothe third embodiment, the effects similar to those of the evaluationsystem 1, the evaluation device 20, the evaluation method, theevaluation program, and the recording medium according to the firstembodiment are exerted. Moreover, according to the evaluation system 1B,the user terminal 10B, the evaluation method, the evaluation program,and the recording medium according to the third embodiment, sincetransmission and reception of data via the network NW does not have tobe carried out, the time lag due to the communication via the network NWdoes not occur, and the speed of response can be improved. Moreover, thetraffic and communication fee of the network NW can be reduced.

Note that the evaluation systems, the evaluation devices, the evaluationmethods, the evaluation programs, and the recording medium according tothe present disclosure are not limited to the above describedembodiments.

For example, in a case in which the correction of the evaluation resultby the user is not carried out, the user terminals 10, 10A, and 10B arenot required to include the corrected information acquisition unit 17.

Also, in the neural network NN, batch normalization may be carried out.The batch normalization is processing to convert output values of layersso that dispersion becomes constant. In such a case, since there is noneed to use the bias value, the nodes (for example, the node 41 b andthe node 421 b), which output bias values, can be omitted.

Also, the evaluation units 18 and 22 may evaluate the coverage based onthe evaluation image by using a method other than the neural network.

Also, the output unit 16 may output the evaluation result to a memory(storage device) not shown and save the evaluation result in the memory.The output unit 16, for example, creates management data, in which amanagement number capable of uniquely identifying the evaluation result,the date on which the evaluation is carried out, etc. are associatedwith the evaluation result, and saves the management data.

The shape of the marker MK is not limited to square. The shape of themarker MK may be rectangular.

In the above described embodiments, the marker MK has a shape with whichthe direction of the marker MK can be specified. However, the shape ofthe marker MK is not limited to the shape having directionality. Theshape of the marker MK may be a non-directional shape. For example, asshown in FIG. 18 (a), the shape of a region Rb may be square, and theshape of a region Rw may be a square slightly smaller than the regionRb. The regions may be disposed so that the center point of the regionRb and the center point of the region Rw are overlapped with each otherand that the sides of the region Rb and the sides of the region Rw areparallel to each other. If the marker MK has a non-directional shape,creation of the marker MK can be simplified since the shape of themarker MK is simple. Moreover, since the direction of the marker MK isnot important, the user can easily capture an image of the evaluationtarget.

As shown in FIG. 18(b), the marker MK may have an opening Hm. Theopening Hm is a through hole, which penetrates through a sheet-likemember on which the marker MK is drawn. The opening area of the openingHm is sufficiently larger than the area of the evaluation region Re,which can be extracted. Therefore, the correction units 13, 24 mayextract the region exposed via the opening Hm from the captured image aspreprocessing of extraction of the evaluation region Re. Then, thecorrection units 13, 24 may extract the evaluation region Re from theextracted region based on the size of the dent region De included in theextracted region.

If the marker MK not surrounded by the frame F2 is used, the boundarybetween the marker region Rm and the region of the evaluation targetsometimes becomes obscure due to reflection of light, etc. In such acase, an edge cannot be detected in the edge detection processing insome cases. In object detection, erroneous detection increases if adetermination threshold value is too low, and missed detection increasesif the determination threshold value is too high. Moreover, in theobject detection per se, the direction (angle) of the marker region Rmcannot be obtained. Furthermore, in a case in which: the marker regionRm is extracted by object detection processing, then edge enhancementprocessing is carried out, and edge detection processing is furthercarried out, detection accuracy improves, but missed detection may occurif the color of the outer edge part of the marker region Rm and thecolor of the periphery of the marker region Rm are almost the same.

On the other hand, regarding the markers MK shown in FIGS. 6(b) to 6(f)and FIGS. 18(c) and 18(d), the marker MK is surrounded by the frame F2,and the gap Rgap is provided between the frame F2 and the region Rb. Thegap Rgap surrounds the region Rb along the edge F1. The color of the gapRgap is different from the color of the outer edge part (morespecifically, the region Rb) of the marker MK. Therefore, even if thecolor of the periphery (outside of the frame F2) of the marker region Rmis similar to the color of the outer edge part (region Rb) of the markerregion Rm, the outer edge of the marker region Rm can be detected sincethe outer edge (edge F1) of the marker region Rm is clear. For example,in a case in which: the marker region Rm is extracted in objectdetection processing, edge enhancement processing is then carried out,and edge detection processing is further carried out, the vertexes(vertexes Pm1 to Pm4) of the region Rb can be more reliably detected.Therefore, the marker region Rm can be extracted at high speed with highaccuracy. As a result, the evaluation accuracy of the coverage can befurther improved. Note that the distance (the width of the gap Rgap)between the frame F2 and the region Rb may be equal to or more than onetenth of one side of the marker MK, for example, in order to ensure thegap Rgap. The distance (the width of the gap Rgap) between the frame F2and the region Rb may be equal to or less than half of one side of themarker MK, for example, in consideration of usability of the marker MK.

Meanwhile, as shown in FIGS. 18(c) and 18(d), the frame F2 is notrequired to be a frame which completely surrounds the marker MK. Inother words, the frame F2 may be provided with a noncontinuous part(s)Fgap. For example, the frame F2 is not limited to a solid line, but maybe a broken line. In such a case, the frame F2 has a shape in which theframe line of the frame F2 is interrupted in the middle. If the frame F2is provided with the noncontinuous parts Fgap, detection accuracy of themarker region Rm improves since the possibility that the regionsurrounded by the frame F2 is detected as the marker region Rm in theedge detection processing, etc. can be reduced. In other words, thevertexes of the marker region Rm (region Rb) can be more reliablydetected since the possibility that the vertexes of the frame F2 aredetected can be reduced. As a result, the evaluation accuracy of thecoverage can be further improved.

As shown in FIG. 19, when the evaluation region Re having the size setbased on the dent region De is to be extracted from the captured imageG, the correction units 13, 24 may randomly determine the evaluationregion Re in the captured image G and extract the determined evaluationregion Re therefrom. In such a case, first, the correction units 13, 24obtain the maximum values of the coordinate, which is possible for areference point Pr of the evaluation region Re. The reference point Pris one of the four vertexes of the evaluation region Re and, in thiscase, is the vertex which is the closest to the origin of X-Y coordinateamong the four vertexes of the evaluation region Re. For example, if thelength of one side of the evaluation region Re is 100 pixels, theX-coordinate maximum value x_(crop_max) and the Y-coordinate maximumvalue y_(crop_max) of the reference point Pr are expressed by followingFormula (2). Note that a vertex Pg1 of the captured image G ispositioned at the origin (0, 0), a vertex Pg2 is positioned at (X_(g),0), a vertex Pg3 is positioned at (X_(g), Y_(g)), and a vertex Pg4 ispositioned at (0, Y_(g)).

[Formula 2]

(x _(crop_max) ,y _(crop_max))=(X _(g)−100,Y _(g)−100)  (2)

The correction units 13, 24 randomly determine the coordinate (X_(crop),Y_(crop)) of the reference point of the evaluation region Re by usingFormula (3). Note that the function random (minimum value, maximumvalue) is a function which returns an arbitrary value included in therange from the minimum value to the maximum value.

[Formula 3]

x _(crop) ,y _(crop))=(random(0,x _(crop_max)),random(0,y_(crop_max)))  (3)

If the deter mined evaluation region Re and the marker region Rm areoverlapped with each other, the correction units 13, 24 may determinethe coordinate of the reference point of the evaluation region Re again.

As shown in FIG. 20, the correction units 13, 24 may extract theevaluation region Re from the captured image G by specifying anextracting direction with respect to the marker region Rm. In such acase, first, the correction units 13, 24 calculate the coordinate(x_(cg), y_(cg)) of a center position Cg of the captured image G and thecoordinate (x_(cm), y_(cm)) of a center position Cm of the marker regionRm. Then, the correction units 13, 24 calculate a vector V, which isfrom the center position Cm to the center position Cg, as shown inFormula (4).

[Formula 4]

V=(x _(cg) −x _(cm) ,y _(cg) −y _(cm))=(x _(v) ,y _(v))  (4)

The correction units 13, 24 determine the position of the evaluationregion Re in the direction indicated by the vector V from the markerregion Rm. The correction units 13, 24 determine the position of theevaluation region Re, for example, so that the reference point Pr of theevaluation region Re is positioned in the direction indicated by thevector V from the center position Cm. In this case, the reference pointPr is the vertex closest to the marker region Rm among the four vertexesof the evaluation region Re. The correction units 13, 24 determine theposition of the evaluation region Re, for example, so that theevaluation region Re is not overlapped with the marker region Rm.Specifically, the correction units 13, 24 calculate the coordinate(X_(crop_max), y_(crop_max)) of a reference point Pr_max farthest fromthe marker region Rm among the possible coordinates of the referencepoint Pr, and the coordinate (x_(crop_min), y_(crop_min)) of a referencepoint Pr_min closest to the marker region Rm among them. Then, thecorrection units 13, 24 determine the position of the evaluation regionRe so that the reference point Pr is positioned on the line segmentbetween these two points.

REFERENCE SIGNS LIST

-   1, 1A, 1B Evaluation system-   10, 10A, 10B User terminal-   11 Image acquisition unit-   13, 24 Correction unit-   16 Output unit-   17 Corrected information acquisition unit-   18, 22 Evaluation unit-   20, 20A Evaluation device-   21 Reception unit (Image acquisition unit)-   23 Transmission unit (output unit)-   De Dent region-   G Captured image-   NN Neural network-   Re Evaluation region

1: An evaluation system configured to evaluate coverage of an evaluationtarget by using a captured image of the evaluation target, theevaluation system comprising: an image acquisition unit configured toacquire the captured image; a correction unit configured to generate anevaluation image by correcting the captured image; an evaluation unitconfigured to evaluate the coverage based on the evaluation image; andan output unit configured to output a result of the evaluation carriedout by the evaluation unit, wherein the correction unit extracts anevaluation region from the captured image based on a size of a dentregion included in the captured image and generates the evaluation imagebased on the evaluation region, and the dent region is an image of adent formed on the evaluation target. 2: The evaluation system accordingto claim 1, wherein the correction unit extracts the evaluation regionfrom the captured image so that larger the size of the dent region is,the evaluation region becomes larger. 3: The evaluation system accordingto claim 2, wherein the correction unit sets the size of the evaluationregion by multiplying the size of the dent region by a constantdetermined in advance and extracts the evaluation region from thecaptured image. 4: The evaluation system according to claim 1, whereinthe correction unit expands or contracts the evaluation region so as toadjust the size of the dent region to a predetermined size. 5: Theevaluation system according to claim 1, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 6: Theevaluation system according to claim 1, wherein the correction unitremoves specular reflection from the evaluation region. 7: Theevaluation system according to claim 1, wherein the evaluation unitevaluates the coverage by using a neural network. 8: An evaluationdevice configured to evaluate coverage of an evaluation target by usinga captured image of the evaluation target, the evaluation devicecomprising: an image acquisition unit configured to acquire the capturedimage; a correction unit configured to generate an evaluation image bycorrecting the captured image; an evaluation unit configured to evaluatethe coverage based on the evaluation image; and an output unitconfigured to output a result of the evaluation carried out by theevaluation unit, wherein the correction unit extracts an evaluationregion from the captured image based on a size of a dent region includedin the captured image and generates the evaluation image based on theevaluation region, and the dent region is an image of a dent formed onthe evaluation target. 9: An evaluation method of evaluating coverage ofan evaluation target by using a captured image of the evaluation target,the evaluation method comprising: a step of acquiring the capturedimage; a step of generating an evaluation image by correcting thecaptured image; a step of evaluating the coverage based on theevaluation image; and a step of outputting a result of the evaluationcarried out in the step of evaluating the coverage, wherein, in the stepof generating the evaluation image, an evaluation region is extractedfrom the captured image based on a size of a dent region included in thecaptured image, and the evaluation image is generated based on theevaluation region, and the dent region is an image of a dent formed onthe evaluation target. 10-11: (canceled) 12: The evaluation systemaccording to claim 2, wherein the correction unit expands or contractsthe evaluation region so as to adjust the size of the dent region to apredetermined size. 13: The evaluation system according to claim 3,wherein the correction unit expands or contracts the evaluation regionso as to adjust the size of the dent region to a predetermined size. 14:The evaluation system according to claim 2, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 15: Theevaluation system according to claim 3, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 16: Theevaluation system according to claim 4, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 17: Theevaluation system according to claim 12, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 18: Theevaluation system according to claim 13, wherein the correction unitcorrects a color of the evaluation region based on a color of areference region included in the captured image, and the referenceregion is an image of a reference having a specific color. 19: Theevaluation system according to claim 2, wherein the correction unitremoves specular reflection from the evaluation region. 20: Theevaluation system according to claim 3, wherein the correction unitremoves specular reflection from the evaluation region. 21: Theevaluation system according to claim 4, wherein the correction unitremoves specular reflection from the evaluation region. 22: Theevaluation system according to claim 5, wherein the correction unitremoves specular reflection from the evaluation region.